Universal Access To All Knowledge
Home Donate | Store | Blog | FAQ | Jobs | Volunteer Positions | Contact | Bios | Forums | Projects | Terms, Privacy, & Copyright
Search: Advanced Search
Anonymous User (login or join us)
Upload

Reply to this post | See parent post | Go Back
View Post [edit]

Poster: Anamon Date: Jan 6, 2012 8:40pm
Forum: web Subject: Re: Honoring present instead of past robots.txt is illogical

It doesn't seem to be the case in absolutely every instance. I remember researching at least one site over the past few weeks whose current owners disallowed IA archiving, but the website that was previously located under the same domain could be opened in the Wayback Machine. Unfortunately I don't remember which site it was at the top of my head, but it was there.

Maybe the IA *does* employ some method to determine when a site has changed ownership and a specific robots.txt does not apply anymore, but it is not terribly reliable.